Combining video data streams of differing dimensionality for concurrent display

ABSTRACT

Embodiments of the invention are generally directed to combining video data streams of differing dimensionality for concurrent display. An embodiment of an apparatus includes an interface to receive multiple video data streams, a dimensionality of each video stream being either two-dimensional (2D) or three-dimensional (3D). The apparatus further includes a processing module to process a first video data stream as a main video image and one or more video data streams as video sub-images, the processing module including a video combiner to combine the main video data stream and the sub-video data streams to generate a combined video output. The processing module is configured to modify a dimensionality of each of the video sub-images to match a dimensionality of the main video image.

TECHNICAL FIELD

Embodiments of the invention generally relate to the field of electronicimage display and, more particularly, combining video data streams ofdiffering dimensionality for concurrent display.

BACKGROUND

A display system, such as a television, a computer, or other similardisplay system, may be utilized to generate a display of multiple videoimages, the images being generated from multiple video data streams. Thedisplay may include concurrent display of multiple data streams.

In particular, a display system may generate a main image and one ormore sub-images. For example, a Picture in Picture (PiP) display is afeature of certain video transmitter and receiver elements. In a PiPdisplay, a first channel (main image) is displayed using the majority ofthe display (such as a full screen display) at the same time as one ormore other channels (sub-images) are displayed in inset windows. Thus,the one or more sub-images generally obscure a portion of the mainimage.

However, video technology is evolving and, rather than being simplytwo-dimensional (2D) images, may include three-dimensional (3 D) images.In an example, data may include 2D HDMI™ (High Definition MultimediaInterface) video data streams as well as 3D HDMI video data streams.(High Definition Multimedia Interface 1.4 Specification, issued May 28,2009) Thus, data streams received for generation of images may be 2Dvideo data streams, 3D video data streams, or a combination of 2D and 3Dvideo data streams.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings inwhich like reference numerals refer to similar elements.

FIG. 1 is an illustration of systems to display 2D and 3D video datastreams;

FIG. 2 is an illustration of 2D and 3D video data frames;

FIG. 3 is an illustration of an embodiment of an apparatus and systemfor processing and display of main video and sub-video data streams;

FIG. 4 is a flowchart to illustrate an embodiment of a process forhandling video data streams; and

FIG. 5 illustrates an embodiment for combining a 2D main video datastream and a 2D sub-video data stream;

FIG. 6 illustrates an embodiment for combining a 3D main video datastream and a 3D sub-video data stream;

FIG. 7 illustrates an embodiment for combining a 2D main video datastream and a 3D sub-video data stream;

FIG. 8 illustrates an embodiment for combining a 3D main video datastream and a 2D sub-video data stream;

FIG. 9A illustrates an embodiment for shifting 2D sub-video data streamswithin a 3D main video data stream;

FIG. 9B illustrates an embodiment for shifting 3D sub-video data streamswithin a 3D main video data stream;

FIG. 10 illustrates an embodiment of a video combiner for combining datastreams of varying dimensionality; and

FIG. 11 illustrates an embodiment of an apparatus or system forprocessing data streams of varying dimensionality.

SUMMARY

Embodiments of the invention are generally directed to combining videodata streams of differing dimensionality for concurrent display.

In a first aspect of the invention, an embodiment of an apparatusincludes an interface to receive multiple video data streams, adimensionality of each video stream being either two-dimensional (2D) orthree-dimensional (3D). The apparatus further includes a processingmodule to process a first video data stream as a main video image andone or more video data streams as video sub-images, the processingmodule including a video combiner to combine the main video data streamand the sub-video data streams to generate a combined video output. Theprocessing module is configured to modify a dimensionality of each ofthe video sub-images to match a dimensionality of the main video image.

In a second aspect of the invention, an embodiment of a method includesreceiving multiple video data streams, a dimensionality of each of thevideo data streams being either two-dimensional (2D) orthree-dimensional (3D). A first video data stream is selected as a mainvideo channel, and one or more video data streams are selected as asub-video channels. The dimensionality of each of the sub-video datastreams is converted to match the dimensionality of the first datastream. A combined video output is generated, the video output includinga main video image generated from the main video channel and a videosub-images generated from the sub-video channels.

In a third aspect of the invention, an embodiment of a video combinerincludes a multiplexer to multiplex a main video data stream with one ormore sub-video data streams to generate combined pixel data, wherein thedata streams may be either three-dimensional (3D) or two-dimensional(2D). The video combiner further includes a synchronization extractor toextract synchronization signals from the main video data stream, a firstcoordinate processor to identify pixels to be included in the combinedpixel data based on the extracted synchronization signals, where thefirst coordinate processor operates for 2D and 3D main video streams,and a 3D video module including a second coordinate processor toidentify pixels to be included in the combined pixel data based on theextracted synchronization signals, where the second coordinate processoroperates for 3D main video streams.

DETAILED DESCRIPTION

Embodiments of the invention are generally directed to combining videodata streams of differing dimensionality for concurrent display.

In some embodiments, a method, apparatus, or system is provided forconcurrent display of multiple video data streams, where the video datastreams may include streams of differing dimensionality. The datastreams may include both two-dimensional (2D) and three-dimensional (3D)data streams. As used herein, the dimensionality of an image or videostream refers to type or number of dimensions represented by the imageor video, and thus whether the video or image is of 2D or 3Ddimensionality.

In some embodiments, a method, apparatus, or system may operate tocombine or mix images generated from video data streams such that one ormore sub-video images are displayed with a main video image in acombined video output, where the method, apparatus, or system operatesto match the dimensionality of the images. In some embodiments, one ormore sub-video images are converted or synthesized to match thedimensionality of such sub-video images with a main video image.

FIG. 1 is an illustration of systems to display 2D and 3D video datastreams. In this illustration, for a two-dimensional case, atransmitting device (data source) 105, such as an HDMI transmitter, mayprovide a data stream 110 comprising a stream of 2D data frames. The 2Ddata stream is received by a receiving device (data sink) 115, such as ahigh definition television (HDTV), to decode and display the 2D image asthe data stream is received. 3D video format is a newer feature of HDMI,in which the viewer is to see a slightly different image in each eye tocreate an illusion of depth in an image. For a three-dimensional case, atransmitting device for 3D video 120, such as a 3D-compatible HDMItransmitter, may provide a data stream 125 comprising a stream of 3Ddata frames containing left and right channel images. As shown in FIG.1, 3D-capable HDMI transmitters pack both left and right images within asingle frame for transmitting the frames over an HDMI data stream. The3D data stream is received by a receiving device 130, such as an HDTVwith 3D video capability, to display the left and right channels. Whenthe 3D-capable HDTV 130 receives a 3D frame, it decodes and splits adata frame into left and right images. There are several methods fordisplaying stereoscopic image, with active shutter glasses 140 being apopular method for HDTV viewing. As illustrated, the HDTV implementsstereoscopic display by alternating between left and right image on adisplay panel. In this illustration, active glasses 140 block or passlight in sync with the left image 150 or right image 145 being displayedby the HDTV 130, with the HDTV 130 including a sync emitter 135 tobroadcast a synchronization signal for operation of the active glasses140.

FIG. 2 is an illustration of 2D and 3D video data frames. FIG. 2illustrates a 2D video format 205 and a 3D video format 210. In the 2Dformat 205, a single active video region is provided, such as Frame 1followed by second frame, Frame 2. In the 3D video format 210, twoactive video regions, shown as a left region and a right region,together with and active space between the two active video regions,compose a 3D active video frame. There are several possible formats forthe 3D video structure, with the possibilities including frame packing,field alternative, line alternative, side-by-side, L+ depth and others.Most of such formats have a similarity to the illustrated framestructure in that two 2D frames (left and right) comprise a single 3Dframe. However, embodiments are not limited to this type of 3Dstructure.

FIG. 3 is an illustration of an embodiment of an apparatus and systemfor processing and display of main video and sub-video data streams. Inthis illustration, a general flow diagram of an embodiment forgenerating PiP video from multiple video streams is provided. In someembodiments, one of multiple incoming video channels is selected by theviewer to be used for main video. In this illustration, the chosen videochannel is Video 1, element 302, to produce the main video 330. One ormore other incoming video channels may be selected for sub-videochannels, which in this illustration are Video 2, element 304, viasub-channel selection 314 through Video N+1, element 306, viasub-channel selection 316. In some embodiments, a sub-video channel mayinclude an on-screen display (OSD), where an OSD is a feature to overlayinformation, such as, for example, a setup menu or a closed caption,over a video image. In some embodiments, a video channel selected as asub-video channel may be downsized, including but not limited todownsampling, downscaling, or cropping of the video channel, to fit ascreen window for display. In some embodiments, a downsizing element,such as downsampling process or module 318-320, reduces the size of thesub-video coming from the sub-channels, and generates sub-images thatare denoted in FIG. 3 as Sub 1, element 332, and Sub N, element 334. Insome embodiments, in order to synchronize sub-images to the main videostream, the sub-images are received from the downsampling modules andtemporarily stored in one or more buffers 322-324 prior to combiningimages. In some embodiments, the buffers are utilized to provide pixeldata for the sub-video to be overlaid in a portion or portions of themain video 330. In some embodiments, a video combiner process or module340 operates to merge the main video image 330 and sub-video images332-334 for display within a single screen, shown as a combined videooutput 350 containing main video 352 and sub-videos 354-356. In contrastto conventional PiP video that assumes that all incoming video streamsare 2D video that may have different resolution and sampling rates, someembodiments provide for enhancement of video combination function tosupport 2D and 3D video together for PiP display.

In some embodiments, an apparatus, system, or method provides forcombining both homogeneous and heterogeneous video for PiP display. Insome embodiments, for heterogeneous PiP display, at least one of theincoming video data streams is 2D video data while at least one of theincoming video data streams is 3D video data. In some embodiments, anoutgoing video may be either a 2D or 3D video image depending on thedimensionality of the main incoming video.

Table 1 illustrates combinations of incoming 2D and 3D video datastreams and the resulting outgoing PiP video image. In some embodiments,the dimensionality of the outgoing PiP video is associated with thedimensionality of the data stream that is selected to be the main videodata image.

TABLE 1 Incoming Main Video Incoming Sub-Video Outgoing Case Data StreamData Stream PiP Video 2D in 2D 2D 2D 2D 3D in 3D 3D 3D 3D 3D in 2D 2D 3D2D 2D in 3D 3D 2D 3D

FIG. 4 is a flowchart to illustrate an embodiment of a process forhandling video data streams. In this illustration, multiple video inputsare received 405, wherein the video inputs may be any combination of 2Dand 3D video data stream. Of the video inputs, a main channel and one ormore sub-channels are identified 410. In some embodiments, thesub-channels are downsized to form sub-videos for a PiP display 415.

In some embodiments, if the main video is 2D 420 and a sub-channel is 2D425, this then results in a combination of the 2D main video and 2Dsub-video 435, such as occurs in a conventional PiP operation. However,if the main video is 2D 420 and a sub-channel is 3D 425, a 2D sub-videois synthesized from the 3D sub-channel 430. For example, the 2Dsub-channel may be synthesized by choosing either the left channel orthe right channel of the 3D video data stream for the video to bedownsampled and combined to utilize for the PiP video output. The 2Dmain video and 2D synthesized sub-video are combined to form thecombined PiP video output 435. Subsequent to combination, the video maybe presented, with the combined video being the 2D sub-video as apicture in picture over the 2D main video 440.

In some embodiments, if the main video is 3D 420 and a sub-channel is 2D445, then a 3D sub-video is synthesized from the 2D sub-channel 450. Forexample, the 3D sub-video may be synthesized by copying the sub-channelto both left and right sub-channels for the synthesized 3D sub-channel.The synthesized 3D sub-channels are downsized and combined with the 3Dmain channel to generate the PiP video 455. The 3D main video and 3Dsynthesized sub-video are combined to form the combined PiP video output455. If the main video is 3D 420 and a sub-channel is 3D 445, this thenresults in a combination of the 3D main video and 3D sub-video 455. Withthe use of the 3D main video, the combination of the videos may includeshifting the relative viewing distance of the sub-video compared withthe main video 460. Subsequent to combination, the video may bepresented, with the combined video output being the 3D sub-video as apicture in picture over the 3D main video 465.

FIG. 5 illustrates an embodiment for combining a 2D main video datastream and a 2D sub-video data stream. In some embodiments, a 2D videochannel may be selected as a main channel 510, each video frameincluding a single main video frame 530. A 2D video channel may furtherbe chosen as a sub-channel 520, each video frame including a singlevideo frame that is downsized to form the sub-video frame 532. In someembodiments, a video combiner 540 receives the main video frames and thesub-video frames, where the video combiner 540 operates to merge themain video and sub-video streams. In some embodiments, the combiningprocess replaces pixels of the main video with pixels of the sub-videowithin a sub-frame region that is defined by the viewer or the videosystem. The result is a combined video 550 including a main video frame552 and one or more sub-video frames 554, where the one or moresub-video frames obscure a portion of the main video frame 552.

FIG. 6 illustrates an embodiment for combining a 3D main video datastream and a 3D sub-video data stream. In some embodiments, a 3D videochannel may be selected as a main channel 610, each video frameincluding a left video frame region 630 and a right video frame region631. A 3D video channel may further be chosen as a sub-channel 620, eachvideo frame including a left video frame region that is downsized toform the left sub-frame region 632 and a right video frame region thatis downsized to form the right sub-frame region 633. Thus, both the mainvideo and sub-video are 3D video that contain left and right regionswithin a single 3D frame. In some embodiments, a video combiner 640receives the main video frames and the sub-video frames and operates tomerge the main video and sub-video streams, the video combiner 640inserting the left region of sub-video into the left region of mainvideo and inserts the right region of the sub-video into the rightregion of main video. The combined video 650 includes a main left videoregion 652 with left sub-video region 654 and a main right video region653 with right sub-video region 655, where sub-video regions obscure aportion of the main video regions.

FIG. 7 illustrates an embodiment for combining a 2D main video datastream and a 3D sub-video data stream. In this case, a generated PiPvideo is 2D and does not present 3D effects on viewer's screen. In someembodiments, because the main video is 2D, the dimensionality of thegenerated PiP video will be 2D. However, the incoming sub-video is 3D.In some embodiments, in order to match the sub-video dimensionality tothe main video dimensionality, the 3D sub-video is synthesized togenerate 2D video. There are multiple methods for converting orsynthesizing 3D video to form 2D video. In some embodiments, a methodfor conversion includes discarding one side of a video region and usingonly the other side. For example, the video combiner may discard theleft region of each sub-video frame and insert the right region into themain video. Although viewers can only see the right image of eachsub-video frame in the generated inset screen, there generally is nomajor loss of information because only a slight difference between theleft and right images is required for creating the illusion of depth inthe image. However, embodiments are not limited to any particularprocess for converting a 3D channel to generate a 2D video image.

As illustrated in FIG. 7, a 2D video channel may be selected as a mainchannel 710, each video frame including a single video frame region 730.A 3D video channel may be chosen as a sub-channel 720, each video frameincluding a left video frame region that is downsized to form the leftsub-frame region 732 and a right video frame region that is downsized toform the right sub-frame region 733. Thus, the main video channel andthe sub-video channel have differing dimensionalities. In someembodiments, the dimensionality of the sub-video is converted togenerate a 2D sub-video image. In some embodiments, a video combiner 740receives the main video frames and the sub-video frames and operates tomerge the main video and sub-video streams, the video combinereliminating either the left or the right region on the sub-video andinserting the remaining region of sub-video into the main video. Thecombined video 750 includes a main video region 752 with an inset windowcontaining the right or left sub-video region 754.

FIG. 8 illustrates an embodiment for combining a 3D main video datastream and a 2D sub-video data stream. In this illustration, an incomingsub-video is 2D and thus there is only a single image per 2D frame, asopposed to one left image and one right image per 3D frame. In someembodiments, in order to match the dimensionality of the sub-channel 820(which is downsized to generate the sub-video image 832) to the 3Dformat of main video channel 810 (for which each frame includes a leftregion 830 and right region 831) the video combiner 840 synthesizes a 3Dvideo image from the 2D sub-video data. In some embodiments, the videocombiner 840 operates to insert the same sub-image twice into the mainvideo frame. In some embodiments, a copy of the same image of thesub-video is inserted into both the left region and the right region ofthe main video image to generate the PiP video 850, illustrated as firstsub-video 854 in left region 852 and second sub-video 855 in rightregion 853.

However, a viewer viewing the generated PiP video in FIG. 8 will see the3D effects of the main video outside the inset window while not seeingany 3D effect inside the inset window. Thus, if the inset window is notmodified, the image inside the inset window may appear to be flat andtwo-dimensional at the same depth as the frame of the display screen.Thus, in some embodiments, a synthesized inset video sub-image isfurther modified to change the apparent depth of the video.

FIG. 9A illustrates an embodiment for shifting 2D sub-video data streamswithin a 3D main video data stream. In this illustration, an optionaland supplemental method for enhancing “2D in 3D” PiP video is provided.Although a 3D effect inside an inset window in “2D in 3D” PiP video isnot generated because the source of the sub-video data stream does notinclude 3D information, in some embodiments an apparatus or system mayadjust the apparent “depth” of the entire inset window. The term “depth”here indicates a virtual distance that a viewer perceives when theviewer views the screen with 3D glasses.

When the video combiner inserts a sub-image to the same location forboth left and right regions of a main video as depicted in video 910 inFIG. 9A, the inset window appears to the viewer to be located in thesame distance as the frame of the screen. In some embodiments, theapparent depth for the viewer may be adjusted such that the inset windowappears to the viewer to be located deeper/further away than the frameof the screen. As shown in video 920, the video combiner may locate thesub-image more left in the left region and places the same sub-imagemore right in the right region. The offset between the two sub-images isindicated by the symbol “Δ”. As the value of Δ becomes larger, viewerperceives that the inset window is located deeper than (or farther awayfrom) the frame of the screen.

In some embodiments, an apparatus or system may also adjust the depth ofan inset window such that the viewer perceives that the inset windowpops up from the screen. As illustrated in video 930, a video combinermay place the sub-image more right in the left region and place the samesub-image more left in the right region. The offset between twosub-images is indicated by the symbol “−Δ”. As the value of Δ becomesmore negative (below zero), the viewer perceives that the inset windowpops up more (or thus is located nearer to the viewer) than the frame ofthe screen.

FIG. 9B illustrates an embodiment for shifting 3D sub-video data streamswithin a 3D main video data stream. In this illustration, an insetwindow in video 940 already has 3D effects, the inset window being basedon a 3D video data stream. In some embodiments, an apparatus or systemfurther provides an offset to generate a perceived depth to insetwindows. In this illustration, when a positive value of Δ is provided,as shown in the shift of the inset windows in video 950, the viewerperceives that the entire 3D image within the inset window is locateddeeper (is farther away) than the main video. In some embodiments,similarly, when a negative value of Δ is provided, as illustrated invideo 960, the viewer perceives that the entire 3D image with the insetwindow pops up from (is closer than) the main video.

In some embodiments, the depth adjustment feature may be utilized toallow viewers to focus on the major object. The major object may beeither the main video or the one or more inset windows. For example, innormal picture-in-picture mode, viewers typically want to focus on themain video. If the inset window pops up or is located in the same depthas the frame of the screen, the inset windows may distract viewers'focus and concentration. In this example, an apparatus or system maylocate the inset windows deeper by setting the value of Δ to a positivevalue so that viewers can focus more on the main video. In anotherexample, in a channel switching mode viewers want to navigate using theinset windows to select the next channel to watch. In this case, viewersmay prefer to focus on the inset windows. As shown in video 930 or 960,an apparatus may adjust the depth of inset windows to pop up by using anegative Δ value, and thus operate to attract a viewer's attention.Thus, in some embodiments, if the major object is the main video, thevideo combiner may utilize a positive Δ value to increase the perceiveddepth of the inset windows, and if the major object is an inset video,the video combiner may utilize a negative Δ value to decrease theperceived depth of the inset windows. In some embodiments, the depthadjustment feature may be further utilized to adjust an apparent depthof an on-screen display (OSD). An OSD may be treated as a sub-videochannel for purposes of adjusting the depth of the OSD as illustrated inFIGS. 9A and 9B. Utilizing the depth adjustment feature, a system cancause the OSD to appear to pop up from or locate deeper than the mainvideo.

FIG. 10 illustrates an embodiment of a video combiner for combining datastreams of varying dimensionality. The video combiner 1000 may be, forexample, video combiner 340 illustrated in FIG. 3. In FIG. 10 a videocombiner 1000 operates to take main video channel and one or moresub-video channels as inputs and generates an outgoing PiP video. Insome embodiments, the video combiner operates to forward the data framesof the main video stream with minimal modification, and then replacespixels of main video within an inset window with pixels of a sub-videoto form the resulting PiP image.

In some embodiments, the video combiner 1000 includes multiple modulesas shown in FIG. 10. In some embodiments, the video combiner receivesmultiple video channels 1005, including a channel chosen as the mainvideo and one or more other channels that may be chosen as sub-videos,such as Sub 1 through Sub N in FIG. 10. In some embodiments, the videochannels 1005 are received by a multiplexer 1040 that operates toreplace pixel values of the main channel with pixels of the sub-channelsto generate a PiP display. In some embodiments, the multiplexer mayutilize alpha-blending to mix the main channel pixel data andsub-channel data pixel in a pre-defined ratio, where alpha-blendingdescribes a process for combining a first (alpha) image with one or moreimage layers to provide a translucent image. In some embodiments, a syncextract module operates to separate synchronization signals such asVsync (Vertical synchronization), Hsync (Horizontal synchronization) andDE (Data enable) signals from the main video interface. In someembodiments, the synchronization signals 1050 from the main video areforwarded to a synchronization merge process or module 1060 for thegeneration of a PiP video output. In some embodiments, a firstcoordinate processor 1025 traces the coordinate of the currenttransmitted pixel and determines if the current pixel is located insidethe inset window or not. The result of the determination is used tocontrol the multiplexer in selecting the source for the generation ofthe PiP video. In some embodiments, a module for 3D main video 1015includes a vertical synchronization inserter (Vsync inserter) 1020 and asecond coordinate processor 1030. In some embodiments, the secondcoordinate processor 1030 is used in addition to the first coordinateprocessor 1025 when main video is 3D video. In the case of the 3D mainvideo, first coordinate processor 1025 controls the left region of the3D format while second coordinate processor 1030 controls the rightregion of the 3D format. In some embodiments, the second coordinateprocessor 1030 may be shared with the first coordinate processor 1025.In some embodiments, the Vsync inserter 1020 operates to insert anadditional Vsync signal into the active space region in the 3D format ofthe main video, which allows second coordinate processor 1030 tocalculate coordinates without requiring knowledge of the 3D format. Insome embodiments, the resulting pixel values 1055 from the multiplexer1040 together with the sync signal 1050 from the sync extract module1010 are received by the sync merge module 1060 to generate the PiPvideo 1070 for display.

As illustrated in FIG. 9A and FIG. 9B, in circumstances in which thechosen main video channel is a 3D channel, the apparent depth of theinset window may be adjusted by a variance between left and rightsub-images. In this case, the horizontal coordination of firstcoordinate processor 1025 differs from that of second coordinateprocessor 1030. This difference makes the horizontal distance Δ betweenthe left inset window and right inset window, as shown in videos 920 and930 of FIG. 9A and videos 950 and 960 of FIG. 9B.

FIG. 11 illustrates an embodiment of an apparatus or system forprocessing data streams of varying dimensionality. In this illustration,certain standard and well-known components that are not germane to thepresent description are not shown. Under some embodiments, a device orsystem 1100 is an apparatus or system to generate and display concurrentvideo images, the video images being main video images and one or moresub-video images.

Under some embodiments, the apparatus or system 1100 comprises aninterconnect or crossbar 1105 or other communication means fortransmission of data. The data may include audio-visual data and relatedcontrol data. The apparatus or system 1100 may include a processingmeans such as one or more processors 1110 coupled with the interconnect1105 for processing information. The processors 1110 may comprise one ormore physical processors and one or more logical processors. Further,each of the processors 1110 may include multiple processor cores. Theinterconnect 1105 is illustrated as a single interconnect forsimplicity, but may represent multiple different interconnects or busesand the component connections to such interconnects may vary. Theinterconnect 1105 shown in FIG. 11 is an abstraction that represents anyone or more separate physical buses, point-to-point connections, or bothconnected by appropriate bridges, adapters, or controllers. Theinterconnect 1105 may include, for example, a system bus, a peripheralcomponent interconnect (PCI) or PCI express (PCIe) bus, a HyperTransportor industry standard architecture (ISA) bus, a small computer systeminterface (SCSI) bus, a IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, sometimes referred to as“Firewire”. (“Standard for a High Performance Serial Bus” 1394-1995,IEEE, published Aug. 30, 1996, and supplements) The apparatus or system1100 further may include a serial bus, such as a universal serial bus(USB), to which may be attached one or more USB compatible connections.

In some embodiments, the apparatus or system 1100 further comprises arandom access memory (RAM) or other dynamic storage device as a memory1115 for storing information and instructions to be executed by theprocessors 1110. Memory 1115 also may be used for storing data for datastreams or sub-streams. RAM memory includes, for example, dynamic randomaccess memory (DRAM), which requires refreshing of memory contents, andstatic random access memory (SRAM), which does not require refreshingcontents, but at increased cost. DRAM memory may include synchronousdynamic random access memory (SDRAM), which includes a clock signal tocontrol signals, and extended data-out dynamic random access memory (EDODRAM). In some embodiments, memory of the system may contain certainregisters, buffers, or other special purpose memory. The apparatus orsystem 1100 also may comprise a read only memory (ROM) 1130 or otherstatic storage device for storing static information and instructionsfor the processors 1110. The apparatus or system 1100 may include one ormore non-volatile memory elements 1135 for the storage of certainelements.

In some embodiments, a data storage 1120 may be coupled to theinterconnect 1105 of the apparatus or system 1100 for storinginformation and instructions. The data storage 1120 may include amagnetic disk, an optical disc and its corresponding drive, or othermemory device. Such elements may be combined together or may be separatecomponents, and utilize parts of other elements of the apparatus orsystem 1100. In some embodiments, the data storage may include storageof video data 1125 for presentation on a display.

The apparatus or system 1100 may also be coupled via the interconnect1105 to a display device or element 1140. In some embodiments, thedisplay 1140 may include a liquid crystal display (LCD), a plasmadisplay, or any other display technology, for displaying information orcontent to an end user. In some embodiments, the display 1140 may beutilized to concurrently display multiple images, where the multipleimages include a main video and one or more sub-video image. In someembodiments, the multiple images may be generated from multiple videodata streams received by the apparatus or system 1100, where a firstvideo stream is selected as the main video 1142 and one or more othervideo data streams are selected as sub-video images 1144, where themultiple video data streams may differ in dimensionality. In someembodiments, the processors 1110 may operate to process the receiveddata streams to generate a PiP display for viewing by one or moreviewers 1150. In some embodiments, the data streams selected assub-video images may be converted or synthesized to match thedimensionality of the main video 1142.

In some embodiments, an input device 1160 may be coupled to orcommunicate with the apparatus or system 1100 for communicatinginformation and/or command selections to the processors 1110. In variousimplementations, the input device 1160 may be a remote control,keyboard, a keypad, a touch screen, voice activated system, or otherinput device, or combinations of such devices. In some embodiments, theapparatus or system 1100 may further include a cursor control device1165, such as a mouse, a trackball, touch pad, or other device forcommunicating direction information and command selections to the one ormore processors 1110 and for controlling cursor movement on the display1140.

One or more transmitters or receivers 1170 may also be coupled to theinterconnect 1105. In some embodiments, the apparatus or system 1100 mayinclude one or more ports 1175 for the reception or transmission ofdata. Data that may be received or transmitted may include 3D or 2Dvideo data streams 1180. The apparatus or system 1100 may furtherinclude one or more antennas 1178 for the reception of data via radiosignals. The apparatus or system 1100 may also comprise a power deviceor system 1185, which may comprise a power supply, a battery, a solarcell, a fuel cell, or other system or device for providing or generatingpower. The power provided by the power device or system 1185 may bedistributed as required to elements of the apparatus or system 1100.

In the description above, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the present invention may be practicedwithout some of these specific details. In other instances, well-knownstructures and devices are shown in block diagram form. There may beintermediate structure between illustrated components. The componentsdescribed or illustrated herein may have additional inputs or outputsthat are not illustrated or described. The illustrated elements orcomponents may also be arranged in different arrangements or orders,including the reordering of any fields or the modification of fieldsizes.

The present invention may include various processes. The processes ofthe present invention may be performed by hardware components or may beembodied in computer-readable instructions, which may be used to cause ageneral purpose or special purpose processor or logic circuitsprogrammed with the instructions to perform the processes.Alternatively, the processes may be performed by a combination ofhardware and software.

Portions of the present invention may be provided as a computer programproduct, which may include a computer-readable medium having storedthereon computer program instructions, which may be used to program acomputer (or other electronic devices) to perform a process according tothe present invention. The computer-readable medium may include, but isnot limited to, floppy diskettes, optical disks, CD-ROMs (compact diskread-only memory), and magneto-optical disks, ROMs (read-only memory),RAMs (random access memory), EPROMs (erasable programmable read-onlymemory), EEPROMs (electrically-erasable programmable read-only memory),magnet or optical cards, flash memory, or other type ofmedia/computer-readable medium suitable for storing electronicinstructions. Moreover, the present invention may also be downloaded asa computer program product, wherein the program may be transferred froma remote computer to a requesting computer.

Many of the methods are described in their most basic form, butprocesses may be added to or deleted from any of the methods andinformation may be added or subtracted from any of the describedmessages without departing from the basic scope of the presentinvention. It will be apparent to those skilled in the art that manyfurther modifications and adaptations may be made. The particularembodiments are not provided to limit the invention but to illustrateit.

If it is said that an element “A” is coupled to or with element “B,”element A may be directly coupled to element B or be indirectly coupledthrough, for example, element C. When the specification states that acomponent, feature, structure, process, or characteristic A “causes” acomponent, feature, structure, process, or characteristic B, it meansthat “A” is at least a partial cause of “B” but that there may also beat least one other component, feature, structure, process, orcharacteristic that assists in causing “B.” If the specificationindicates that a component, feature, structure, process, orcharacteristic “may”, “might”, or “could” be included, that particularcomponent, feature, structure, process, or characteristic is notrequired to be included. If the specification refers to “a” or “an”element, this does not mean there is only one of the described elements.

An embodiment is an implementation or example of the invention.Reference in the specification to “an embodiment,” “one embodiment,”“some embodiments,” or “other embodiments” means that a particularfeature, structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments. The various appearances of “an embodiment,”“one embodiment,” or “some embodiments” are not necessarily allreferring to the same embodiments. It should be appreciated that in theforegoing description of exemplary embodiments of the invention, variousfeatures of the invention are sometimes grouped together in a singleembodiment, figure, or description thereof for the purpose ofstreamlining the disclosure and aiding in the understanding of one ormore of the various inventive aspects.

What is claimed is:
 1. An apparatus comprising: an interface to receivea plurality of video data streams, a dimensionality of each video streambeing either two-dimensional (2D) or three-dimensional (3D); and aprocessing module to process a first video data stream of the pluralityof video streams as a main video image and a second video data stream ofthe plurality of data streams as a video sub-image, the processingmodule including a video combiner to combine the main video data streamand the sub-video data stream to generate a combined video output;wherein a dimensionality of the first video data stream is 3D and adimensionality of the second video data stream is 2D, and wherein theprocessing module is configured to modify the 2D dimensionality of thevideo sub-image to 3D to match the 3D dimensionality of the main videoimage; wherein conversion of the second video stream from 2D to 3Dincludes copying each frame of data of the second video data stream intoa left channel region and a right channel region and adjusting anapparent depth for a viewer between the video sub-image and a displayframe by modifying a difference between a position of the frame of thesecond video data stream in the left channel region and a position ofthe frame of the second video data stream in the right region.
 2. Theapparatus of claim 1, wherein the processing module is furtherconfigured to downsize the second video stream to fit the second videostream into a region for the video sub-stream.
 3. The apparatus of claim2, wherein downsizing the second video stream comprises downsampling,downscaling, or cropping the video stream.
 4. The apparatus of claim 1,wherein the second video stream includes an on-screen display (OSD), andwherein conversion of the second video stream includes adjusting anapparent depth for the viewer between the OSD and the video frame. 5.The apparatus of claim 1, further comprising a display screen to displayone or more images, wherein the one or more images may include the mainvideo image and the video sub-image, wherein the video sub-image issmaller than the main video image and obscures a portion of the mainvideo image.
 6. The apparatus of claim 1, wherein the video combinerincludes a multiplexer to multiplex the sub-video data stream with themain video data stream to generate output pixel data.
 7. The apparatusof claim 6, wherein the video combiner includes a module to extractsynchronization signals from the first video data stream.
 8. Theapparatus of claim 7, wherein the video combiner includes a module toreceive the output pixel data and the extracted synchronization signalsto generate the combined video output.
 9. The apparatus of claim 6,wherein the video combiner includes one or more coordinate processors todetermine which pixels of the sub-video data stream and the main videodata stream are to be included in the output pixel data.
 10. A methodcomprising: receiving a plurality of video data streams, adimensionality of each of the plurality of video data streams beingeither two-dimensional (2D) or three-dimensional (3D); selecting a firstvideo data stream of the plurality of video data streams as a main videochannel, a dimensionality of the first video data stream being 3D;selecting a second video data stream of the plurality of data streams asa sub-video channel, a dimensionality of the second video data streambeing 2D; converting the dimensionality of the second video data streamfrom 3D to 2D to match the dimensionality of the first video datastream; and generating a combined video output, the video outputincluding a main video image generated from the main video channel and avideo sub-image generated from the sub-video channel; wherein conversionof the dimensionality of the second video data stream from 2D to 3Dincludes copying each frame of data of the second video data stream intoa left channel region and a right channel region and adjusting anapparent depth for a viewer between the video sub-image and a displayframe by modifying a difference between a position of the frame of thesecond video data stream in the left channel region and a position ofthe frame of the second video data stream in the right region.
 11. Themethod of claim 10, wherein generating the video output includesmultiplexing the first video data stream with the second video datastream.
 12. The method of claim 10, further comprising downsizing framesof the second video data stream such that the sub-video image fits acertain inset window for display.
 13. The method of claim 10, whereinthe sub-video channel includes an on-screen display (OSD), and whereinadjusting the apparent depth of the sub-video channel comprisesmodifying a difference between a position of an OSD left region and aposition of an OSD right region.
 14. The method of claim 10, whereingenerating the combined video output includes extracting synchronizationsignals from the main video channel.
 15. A video combiner comprising: amultiplexer to multiplex a main video data stream with one or moresub-video data streams to generate combined pixel data, wherein the datastreams may be either three-dimensional (3D) or two-dimensional (2D); asynchronization extractor to extract synchronization signals from themain video data stream; a first coordinate processor to identify pixelsto be included in the combined pixel data based on the extractedsynchronization signals, the first coordinate processor to operate for2D and a first region of 3D main video streams; and a 3D video moduleincluding a second coordinate processor to identify pixels to beincluded in the combined pixel data based on the extractedsynchronization signals, the second coordinate processor to operate fora second region of 3D main video streams; wherein a dimensionality of amain video data stream is 3D and a dimensionality of a sub-video datastream is 2D, the 2D dimensionality of the sub-video being converted to3D to match the 3D dimensionality of the main video image, wherein thevideo combiner is to copy each frame of data of the sub-video datastream into a left channel region and a right channel region, anapparent depth for a viewer between the video sub-image and a displayframe being adjusted by modifying a difference between a position of theframe of the second video data stream in the left channel region and aposition of the frame of the second video data stream in the rightregion.
 16. The video combiner of claim 15, wherein the secondcoordinate processor is shared with the first coordinate processor. 17.The video combiner of claim 15, wherein the 3D video module furtherincludes a vertical synchronization inserter, the verticalsynchronization inserter to insert an additional verticalsynchronization signal into an active space region in a 3D format of themain video data stream.
 18. The video combiner of claim 15, wherein thesecond coordinate processor operates without knowledge of the 3D formatof the main video data stream.
 19. The video combiner of claim 15,further comprising a module to receive the combined pixel data and theextracted synchronization signals to generate a combined video output.20. A non-transitory computer-readable storage medium having storedthereon data representing sequences of instructions that, when executedby a processor, cause the processor to perform operations comprising:receiving a plurality of video data streams, a dimensionality of each ofthe plurality of video data streams being either two-dimensional (2D) orthree-dimensional (3D); selecting a first video data stream of theplurality of video data streams as a main video channel, adimensionality of the first video data stream being 3D; selecting asecond video data stream of the plurality of data streams as a sub-videochannel, a dimensionality of the second video data stream being 2D;converting the dimensionality of the second video data stream from 3D to2D to match the dimensionality of the first video data stream; andgenerating a combined video output, the video output including a mainvideo image generated from the main video channel and a video sub-imagegenerated from the sub-video channel; wherein conversion of thedimensionality of the second video data stream from 2D to 3D includescopying each frame of data of the second video data stream into a leftchannel region and a right channel region and adjusting an apparentdepth for a viewer between the video sub-image and a display frame bymodifying a difference between a position of the frame of the secondvideo data stream in the left channel region and a position of the frameof the second video data stream in the right region.
 21. The medium ofclaim 20, wherein generating the video output includes multiplexing thefirst video data stream with the second video data stream.
 22. Themedium of claim 20, further comprising instructions that, when executedby a processor, cause the processor to perform operations comprising:downsizing frames of the second video data stream such that thesub-video image fits a certain inset window for display.
 23. The mediumof claim 20, wherein the sub-video channel includes an on-screen display(OSD), and wherein adjusting the apparent depth of the sub-video channelcomprises modifying a difference between a position of an OSD leftregion and a position of an OSD right region.
 24. The medium of claim20, wherein generating the combined video output includes extractingsynchronization signals from the main video channel.
 25. The apparatusof claim 1, wherein adjusting the apparent depth between the videosub-image and the display frame includes providing a negative offsetbetween the left image and the right image to make the video sub-imageappear to be closer to the viewer than the display frame when thesub-image is a main object for the viewer.
 26. The apparatus of claim 1,wherein adjusting the apparent depth between the video sub-image and thedisplay frame for a viewer includes providing a positive offset betweenthe left image and the right image to make the video sub-image appear tobe farther from the viewer than the display frame when the main image isa main object for the viewer.